Interactive clinical diagnosis report system

ABSTRACT

Embodiments of the disclosure provide systems and methods for generating a diagnosis report based on a medical image of a patient. The system includes a communication interface configured to receive the medical image acquired by an image acquisition device. The system further includes at least one processor. The at least one processor is configured to detect a medical condition based on the medical image and automatically generate text information describing the medical condition. The at least one processor is further configured to construct the diagnosis report, where the diagnosis report includes at least one image view showing the medical condition and a report view including the text information describing the medical condition. The system also includes a display configured to display the diagnosis report.

CROSS REFERENCE TO RELATED APPLICATION

The present application is a continuation of U.S. application Ser. No.17/013,632, filed Sep. 6, 2020, which is a continuation of U.S.application Ser. No. 16/154,681, filed Oct. 8, 2018, now U.S. Pat. No.10,803,579, which claims the benefits of priority to U.S. ProvisionalApplication No. 62/572,114, filed Oct. 13, 2017. Each application isincorporated herein by reference in its entirety.

TECHNICAL FIELD

The present disclosure relates to a diagnosis report system, and moreparticularly to, an interactive diagnosis report system thatautomatically generates a diagnosis report based on medical images of apatient, allows a user to edit or otherwise interact with the medicalimages, and updates the diagnostic report according to the userinteraction.

BACKGROUND

Radiologists read medical images to detect abnormalities and makediagnosis of diseases. Writing diagnosis report is also part of routinejobs for radiologists/clinicians. The diagnosis report often conformswith report templates and clinical standards. Diagnosis reports areusually prepared by a radiologist to record his diagnosis. Theradiologist may also perform certain measurements, e.g., the size of atumor, and record the measurements in the report. If the radiologistwants to include an image in his report to be illustrative, he typicallyhas to transfer the image between different platforms, and then manuallycopy and paste it into the report. Generating a diagnosis report is thustedious and inefficient. Some improved report systems can transcribespeech to texts to save radiologists' time to type the report. However,transcription introduces an extra step to the process and thusassociates with additional costs and errors.

Computer aided diagnosis (CAD) tools could significantly simplify thediagnosis procedure for radiologists and increase their workthroughputs. For example, the CAD system can automatically detectsuspicious regions in the images and classify/segment those regions forquantitative analysis to understand the implications. This procedurespeeds up lesion detection and the overall diagnosis process, as well asincrease accuracy. The CAD system can also present visualization ofanalysis results (e.g., bounding boxes, object boundary contours,surface rendering, volume rendering, etc.) to a user, e.g., theradiologist. These visualization results with quantitative analysiscould become parts of a diagnosis report. Then radiologists/cliniciansadd their observations and diagnosis result to the reports. However,automatically generating reports from the CAD system is susceptible tomisdiagnosis and lacks radiologists' inputs and decisions.

Embodiments of the disclosure address the above problems by designing aninteractive diagnosis report system taking advantage of theauto-diagnosis of the CAD system as well as radiologists' inputs basedon their experience.

SUMMARY

Embodiments of the disclosure provide a system for generating adiagnosis report based on a medical image of a patient. The systemincludes a communication interface configured to receive the medicalimage acquired by an image acquisition device. The system furtherincludes at least one processor. The at least one processor isconfigured to detect a medical condition based on the medical image andautomatically generate text information describing the medicalcondition. The at least one processor is further configured to constructthe diagnosis report. The diagnosis report includes at least one imageview showing the medical condition and a report view including the textinformation describing the medical condition. The system also includes adisplay configured to display the diagnosis report.

Embodiments of the disclosure also provide a method for generating adiagnosis report based on a medical image of a patient. The methodincludes receiving, by a communication interface, the medical imageacquired by an image acquisition device. The method further includesdetecting, by at least one processor, a medical condition based on themedical image and automatically generating, by the at least oneprocessor, text information describing the medical condition. The methodalso includes constructing, by the at least one processor, the diagnosisreport. The diagnosis report includes at least one image view showingthe medical condition and a report view including the text informationdescribing the medical condition. The method additionally includesdisplaying the diagnosis report on a display.

Embodiments of the disclosure further provide a non-transitorycomputer-readable medium having instructions stored thereon that, whenexecuted by one or more processors, causes the one or more processors toperform a method for generating a diagnosis report based on a medicalimage of a patient. The method includes receiving the medical imageacquired by an image acquisition device. The method further includesdetecting a medical condition based on the medical image andautomatically generating text information describing the medicalcondition. The method also includes constructing the diagnosis report.The diagnosis report includes at least one image view showing themedical condition and a report view including the text informationdescribing the medical condition. The method additionally includesdisplaying the diagnosis report.

It is to be understood that both the foregoing general description andthe following detailed description are exemplary and explanatory onlyand are not restrictive of the invention, as claimed.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a block diagram of an exemplary diagnosis reportgenerating system, according to embodiments of the disclosure.

FIG. 2 illustrates an exemplary user interface of the diagnosis reportgenerating system of FIG. 1 , according to embodiments of thedisclosure.

FIG. 3 illustrates another exemplary user interface of the diagnosisreport generating system of FIG. 1 , according to embodiments of thedisclosure.

FIG. 4 shows a flow chart of an exemplary method for generating adiagnosis report, according to embodiments of the disclosure.

DETAILED DESCRIPTION

Reference will now be made in detail to the exemplary embodiments,examples of which are illustrated in the accompanying drawings. Whereverpossible, the same reference numbers will be used throughout thedrawings to refer to the same or like parts.

FIG. 1 illustrates a block diagram of an exemplary diagnosis reportgenerating system 100, according to embodiments of the disclosure.Consistent with the present disclosure, diagnosis report generatingsystem 100 is configured to generate a diagnosis report based on medicalimages 102 acquired by an image acquisition device 101. Consistent withthe present disclosure, diagnosis report generating system 100 mayreceive medical images 102 from image acquisition device 101.Alternatively, medical images 102 may be stored in an image databasefirst and diagnosis report generating system 100 may receive medicalimages 102 from the image database. In some embodiments, medical images102 may be two-dimensional (2D) or three-dimensional (3D) images. A 3Dimage may contain multiple 2D image slices.

In some embodiments, image acquisition device 101 may acquire medicalimages 102 using any suitable imaging modalities, including, e.g.,functional MRI (e.g., fMARI, DCE-MRI and diffusion MRI), Cone Beam CT(CBCT), Spiral CT, Positron Emission Tomography (PET), Single-PhotonEmission Computed Tomography (SPECT), X-ray, optical tomography,fluorescence imaging, ultrasound imaging, and radiotherapy portalimaging, etc.

For example, image acquisition device 101 may be an MRI scanner. The MRIscanner includes a magnet that surrounds a patient tube with a magneticfield. A patient is positioned on a padded table that can move into thepatient tube. The MRI scanner further includes gradient coils inmultiple directions (e.g., x, y, and z directions) to create a spatiallyvarying magnetic field on top of the uniform magnetic field created bythe magnet. The uniform magnetic field used by the MRI scanner aretypically between 0.2 T-7 T, e.g., around 1.5 T or 3T. The MRI scanneralso includes RF coils to excite the tissues inside the patient body andtransceivers to receive electromagnetic signals generated by the tissueswhile returning to an equilibrium state.

As another example, image acquisition device 101 may be a CT scanner.The CT scanner includes an X-ray source that emits X-rays against bodytissues and a receiver that receives the residual X-rays afterattenuated by the body tissues. The CT scanner also includes rotatingmechanism to capture X-ray images at different view angles. Suchrotating mechanism can be a rotating table that rotates the patient, ora rotating structure that rotates the X-ray source and the receiveraround the patient. The X-ray images at different angles are thenprocessed by a computer system to construct a two-dimensional (2D) crosssection image or a three-dimensional (3D) image.

Consistent with some embodiments, diagnosis report generating system 100may further receive patient information 104 from patient database 103.Patient information 104 may be meta data recorded through patientregistration or generated with medical image 102. For example, the metadata may include age and gender of the patient, patient medical history,and family medical history, etc. Patient database 103 may in include avolatile or non-volatile, magnetic, semiconductor, tape, optical,removable, non-removable, or other type of storage device or tangible(i.e., non-transitory) computer-readable medium including, but notlimited to, a ROM, a flash memory, a dynamic RAM, and a static RAM. Insome embodiments, patient database 103 may be located on-site withdiagnosis report generating system 100 or off-site (i.e., remotely).

In some embodiments, as shown in FIG. 1 , diagnosis report generatingsystem 100 may include a communication interface 110, a processor 120, adisplay 130, an I/O interface 140, a memory 150, and a storage 160 and.In some embodiments, diagnosis report generating system 100 may havedifferent modules in a single device, such as an integrated circuit (IC)chip (implemented as an application-specific integrated circuit (ASIC)or a field-programmable gate array (FPGA)), or separate devices withdedicated functions. In some embodiments, one or more components ofdiagnosis report generating system 100 may be located in a cloud, or maybe alternatively in a single location (such as in a computer inside aradiologist's office) or distributed locations. Components of diagnosisreport generating system 100 may be in an integrated device, ordistributed at different locations but communicate with each otherthrough a network (not shown).

Communication interface 110 may send data to and receive data fromexternal systems or devices, such as image acquisition device 101 andpatient database 103, via communication cables, a Wireless Local AreaNetwork (WLAN), a Wide Area Network (WAN), wireless networks such asradio waves, a cellular network, and/or a local or short-range wirelessnetwork (e.g., Bluetooth™), or other communication methods. In someembodiments, communication interface 110 can be an integrated servicesdigital network (ISDN) card, cable modem, satellite modem, or a modem toprovide a data communication connection. As another example,communication interface 110 can be a local area network (LAN) card toprovide a data communication connection to a compatible LAN. Wirelesslinks can also be implemented by communication interface 110. In such animplementation, communication interface 310 can send and receiveelectrical, electromagnetic or optical signals that carry digital datastreams representing various types of information via a network.

Consistent with some embodiments, communication interface 110 mayreceive medical images 102 acquired by image acquisition device 101.Consistent with some embodiments, communication interface 110 may alsoreceive patient information 104 from patient database 103. Communicationinterface 110 may further provide the received medical images 102 andpatient information 104 to storage 160 for storage or to processor 120for processing.

Processor 120 may include any appropriate type of general-purpose orspecial-purpose microprocessor, digital signal processor, ormicrocontroller. Processor 120 may be configured as a stand-aloneprocessor module dedicated to diagnosis report generation.Alternatively, processor 120 may be configured as a shared processormodule for performing other functions unrelated to diagnosis reportgeneration.

As shown in FIG. 1 , processor 120 may include multiple modules, such asa computer aided diagnosis unit 122 (hereafter, “CAD unit 122”), and areport generation unit 124, and the like. These modules (and anycorresponding sub-modules or sub-units) can be hardware units (e.g.,portions of an integrated circuit) of processor 120 designed for usewith other components or software units implemented by processor 120through executing at least part of a program. The program may be storedon a computer-readable medium, and when executed by processor 120, itmay perform one or more functions or operations. Although FIG. 1 showsunits 122-124 both within one processor 120, it is contemplated thatthese units may be distributed among multiple processors located near orremotely with each other.

CAD unit 122 is configured to assist doctors in the interpretation ofmedical images. In some embodiments, CA) unit 122 may process digitalimages for typical appearances and to highlight conspicuous sections,such as possible diseases, in order to offer input to support a decisiontaken by the professional. For example, CAD unit 122 performsinteractive/automated algorithms to detect conspicuous structures in amedical image 102 indicative of a medical condition.

In some embodiments, medical images 102 may be copied to CAD unit 122 ina DICOM-format. Pre-processing may be performed on medical images 102such as filtering to reduce image artifacts or noises, and levelingimage quality, e.g., by adjusting the image's different exposureparameters to increase contrast. CAD unit 122 may further segment theimages to identify different regions of interest (e.g., anatomicalstructures) in the image, e.g. heart, lung, ribcage, blood vessels,possible round lesions. Various segmentation methods may be used,including, e.g., matching with an anatomic databank, or using neuralnetworks trained using sample images. The identified structures may beanalyzed individually for special characteristics.

In some embodiments, CAD unit 122 may perform a fully automatic initialinterpretation and triage of studies into some meaningful categories(e.g. negative and positive). Various classification algorithms may beused, including, e.g., the Nearest-Neighbor Rule (e.g. k-nearestneighbors), Minimum distance classifier, Cascade classifier, NaiveBayesian Classifier, Artificial Neural Network, Radial basis functionnetwork (RBF), Support Vector Machine (SVM) and Principle ComponentAnalysis (PCA), etc. If the detected structures meet certain criteria,CAD unit 122 may highlight them in the image for the radiologist, forexample, using boundary contours or bounding boxes. This allows theradiologist to draw conclusions about the condition of the pathology Insome embodiments, CAD unit 122 may further determine one or moreparameters that quantify the medical condition. For example, theparameters may include size (such as diameter, length, width, depth,etc.), volume, pixel intensities, or the contrast characteristics of atumor.

Accordingly, the output of CAD unit 122 may include object detectionresults (e.g. boundary contours or bounding boxes), segmentations,classification results etc. as well as quantitative analysis results(e.g. size, volume, gray-value distribution etc.) derived from thoseresults. Report generation unit 124 is configured to perform reportgeneration algorithms based on the output of CAD unit 122. In someembodiments, the diagnosis report may include various patient,examination, and diagnosis information. In some embodiments, thediagnosis report may be automatically or semi-automatically generated byreport generation unit 124. In some embodiments, report generation unit124 may generate the diagnosis report interactively with a user 105,e.g., a radiologist.

In some embodiments, report generation unit 124 may generate automatedreport content from meta data contained in patient information 104. Forexample, one part of the report may be composed of patient historyinformation (such as name, gender, age of the patient) and scaninformation (such as imaging modality used to acquire medical image 102,the region scanned, and if any contrast agent is used) derived from thepatient's meta data.

Report generation unit 124 may further generate diagnosis content of thereport based on CAD system results. Report generation unit 124 inferstext information of the report from CAD results, such as the objectdetection/segmentation/classification results. In some embodiments, thetext information may indicate, among other things, the type of thedetected object (i.e. bleeding type cerebral hemorrhage) and theposition of the detected object (i.e. left frontal lobe). In someembodiments, the text information may further indicate results ofquantitative analysis, such as diameters, volumes, and densitydistribution, etc.

In some embodiments, a text dictionary used to produce texts based onthe detection results may be established. For example, the textdictionary may be trained using existing clinical reports. The trainingmay be performed by processor 120 or a separate processor. In thetraining stage, the existing clinical reports may be mined to identifykey words to describe image observations, including object properties,types, object locations, etc. According to some embodiments, based onthe mined information, one or more lookup tables may be created for textquery purpose. For example, Table 1 shows an example of object typelookup table, and Table 2 shows an example of object location lookuptable.

TABLE 1 Example of object type lookup table ID text 0 CerebralHemorrhage 1 Subarachnoid hemorrhage 2 Subdural hematoma . . . . . . . .

TABLE 2 Example of object location lookup table Position Id text 0 LeftFrontal Lobe 1 Right Frontal Lobe 2 Left Back Lobe . . . . . . . .

Based on the lookup tables and detection results provided by CAD unit122, report generation unit 124 retrieves the text from the lookuptables and fills the corresponding sections in the report. It iscontemplated that a lookup method is only one exemplary approach toautomatically generate texts based on CAD detection results. Othermethods can be used, such as a machine learning approach, e.g., sentenceprediction from CAD results).

In some embodiments, the report may also include screenshots of 2D/3Dvisualization containing the detected objects. In some embodiments, thereport may display and discuss the top N detected objects, where N is apredetermined number, e.g., 3, 5, 10, etc. If the number of detectedobjects is less than or equal to N, all detected objects are includedthe report. If the number of detected objects is more than N, alldetected objects are included the report. For each detected object, thequantitative numbers may be computed and displayed in the report.

In some embodiments, report generation unit 124 may utilize the metainformation to improve the diagnosis results provided by CAD unit 122.For example, CAD unit 122 may detect lung nodules and predict nodulemalignancy based on medical image 102. Meta data such as age, smokinghistory, occupations, etc. may be used to improve the malignancyprediction of the nodule.

Processor 120 may render visualizations of user interfaces to displaydata on a display 130. Display 130 may include a Liquid Crystal Display(LCD), a Light Emitting Diode Display (LED), a plasma display, or anyother type of display, and provide a Graphical User Interface (GUI)presented on the display for user input and data display. The displaymay include a number of different types of materials, such as plastic orglass, and may be touch-sensitive to receive commands from the user. Forexample, the display may include a touch-sensitive material that issubstantially rigid, such as Gorilla Glass™, or substantially pliable,such as Willow Glass™.

The visualization may include medical images and analysis resultsgenerated by CAD unit 122 as well as the diagnostic report generated byreport generation unit 124. In some embodiments, the CAD results and thediagnostic report may be displayed side-by-side. For example, FIGS. 2-3illustrate exemplary user interfaces 200/300 of diagnosis reportgenerating system 100 of FIG. 1 , according to embodiments of thedisclosure. User interfaces 200/300 may include a report view 210/310displaying the diagnosis report generated by report generation unit 124,and a CAD view 220/320 displaying images and detection results producedby CAD unit 122. It is contemplated that the relative positions andconfigurations of the views are exemplary only, and may be re-arrangedin other embodiments.

As shown in FIGS. 2-3 , CAD view 220/320 shows medical images renderedin views to show the detected object. For example, CAD view 220 in FIG.2 shows a 3D brain image containing multiple image slices in the bottom,and a selected image slice on the top. A cerebral lesion is highlightedusing a different color as well as marked by a boundary contour. Asanother example, CAD view 320 in FIG. 3 shows a 3D lung image in thebottom, and 3 cross-sectional views (i.e., sagittal, coronal, transverseviews) of image at a selected point. The exemplary image shown in CADview 320 includes multiple lung nodules. The lung nodules arehighlighted, e.g., each by a bounding box, in the respective 3cross-sectional views in CAD view 320. In some embodiments, the 3Dviewer in CAD view 220 presents volume rendering, surface rendering andmulti-planar reconstructions (MPR) and curved MPR (for vesselvisualization).

Report view 210/310 includes a patient information section 212/312displaying patient information derived from meta data, such as name,gender, and age of the patient. According to some embodiments, reportview 210/310 further includes an examination section 214/314 describingthe scan information such as the image modality used to acquire medicalimage 102, the part of the patient body scanned, and any contrast agentused in the scan. In some embodiments, report view 210/310 also includesan impression section 216/316 displaying the diagnosis results, e.g.,indicating a medical condition of the patient, based on the CADanalysis. Report view 210/310 may further include a findings section218. In some embodiments, findings section 218/318 may displayscreenshots of medical images with the detected object highlighted,e.g., by a boundary contour or a bounding box, as shown in CAD view220/320. In some embodiments, findings section 218/318 may furtherinclude texts describing the type of detected object (e.g., cerebralhemorrhage, lung nodules) and its location (e.g., left frontal lobe,etc.).

In some embodiments, findings section 218/318 may also show theparameters that quantify the medical condition, such as the size,volume, and intensity distribution of the detected object, as well asother observations. For example, as shown in findings section 218, thediameter of the detected lesion in the left frontal lobe is measured as2.5 cm. Additional observations include, e.g., that the lesion has aneven density, and that the edema surrounding lesion has a low density.For example, as shown in findings section 318, the longest radius of thedetected lung nodule may be 30 mm, and its volume might be 1000 m³.Additional observations include that the nodule has calcium, spiculationand lobulation, and thus is a suspiciously malignant nodule.

In some embodiments, automatically produced report may not always besufficient for the clinical usage as it lacks aradiologist's/clinician's decision. In some embodiments, diagnosisreport generating system 100 may provide interactive tools in the userinterfaces displayed by display 130, to allow the radiologist/clinicianto edit the CAD results and/or the diagnosis report. Returning to FIG. 1, in some embodiments, user 105 may provide a user interaction 106 viaI/O interface 140. I/O interface 140 may be an input/output device thatis configured to receive user input or provide system output to theuser. For example, I/O interface 140 may be a keyboard, a mouse, a clickbutton, a stylus, a touch-screen, a microphone, or any combinationthereof.

In some embodiments, user interaction 106 may be performed in CAD view220/320 to edit the CAD results. For example, user interaction 106 maybe user 105 selecting an image slice from a 3D image (e.g., shown in CADview 220) or selecting a detected object from an object list (e.g.,selecting a lung nodule from a nodule list 328 shown in CAD view 320).As another example, user interaction 106 may further include user 105manually drawing a ruler/curve/region/point/bounding box/text in theview by using image editing tools 224/324.

If user 105 is satisfied with the editing, he could drag the view(s)shown in CAD view 220/320 into the report, or click a “send” button226/326 to send the images shown in CAD view 220/320 view to report view210/310, to be included in, e.g., findings section 218/318. In someembodiments, both 2D and 3D image editing are supported. In addition,user 105 may click on an “A. I.” button 222/322 to cause CAD unit 122 toperform the quantitative analysis again based on the user edits to theimages, and send the updated parameters to report view 210/310 toupdate, e.g., findings section 218/318. In some embodiments, the userinteractions with CAD view 220/320 may automatically update the contentof the report based on user interaction 106, without user 105 having todrag or click on any button.

In some embodiments, user interaction 106 may also include interactionswith report view 210/310. For example, each of sections 212-218 and312-318 may be editable by user 105. An editable area in the diagnosisreport allows user 105 (e.g., a radiologist/clinician) to add, modify,or delete the report content. In some embodiments, user 105 could removethe automatically generated texts and screenshots, such as by clickingon a cancel button 219/319. In addition, user 105 can edit any textsection in the diagnosis report to provide his independent diagnosisbased on medical images 102. In one embodiment, the text input isthrough a keyboard (one embodiment of I/O interface 140). In anotherembodiment, the text input is through a microphone (one embodiments ofI/O interface 140) and processor 120 may perform a speech to texttranslation to transcribe the audio input into texts.

In some embodiments, user interfaces 200/300 may provide additionalfunctionalities such as preview, print, email or archive. For example,user 105 may double-click on the diagnosis report to finalize it, saveit to memory 150/storage 160, and send it to another department viaemail.

Memory 150 and storage 160 may include any appropriate type of massstorage provided to store any type of information that processor 120 mayneed to operate. Memory 150 and/or storage 160 may be a volatile ornon-volatile, magnetic, semiconductor, tape, optical, removable,non-removable, or other type of storage device or tangible (i.e.,non-transitory) computer-readable medium including, but not limited to,a ROM, a flash memory, a dynamic RAM, and a static RAM. Memory 150and/or storage 160 may be configured to store one or more computerprograms that may be executed by processor 120 to perform functionsdisclosed herein. For example, memory 150 and/or storage 160 may beconfigured to store program(s) that may be executed by processor 120 forCAD analysis and diagnosis report generation.

Memory 150 and/or storage 160 may be further configured to storeinformation and data used by processor 120. For instance, memory 150and/or storage 160 may be configured to store medical images 102acquired by image acquisition system 101, and patient information 104provided by patient database 103. Memory 150 and/or storage 160 may alsostore CAD analysis results generated by CAD unit 122, as wellintermediary data created during the CAD process. Memory 150 and/orstorage 160 may also store various parts of a diagnosis report generatedby report generation unit 124, such as images, tables, and texts, etc.The various types of data may be stored permanently, removedperiodically, or disregarded immediately after each frame of data isprocessed.

FIG. 4 shows a flow chart of an exemplary method 400 for generating adiagnosis report, according to embodiments of the disclosure. Forexample, method 400 may be implemented by diagnosis report generatingsystem 100 in FIG. 1 . However, method 400 is not limited to thatexemplary embodiment. Method 400 may include steps S402-S418 asdescribed below. It is to be appreciated that some of the steps may beoptional to perform the disclosure provided herein. Further, some of thesteps may be performed simultaneously, or in a different order thanshown in FIG. 4 .

In step S402, diagnostic report generating system 100 receives one ormore medical images 102 associated with a patient, e.g., from imageacquisition device 101 or a medical image database. Medical images 102may be 2D or 3D images. Medical images 102 can be of any imagingmodality among functional MRI (e.g., fMRI, DCE-MRI and diffusion MRI),Cone Beam CT (CBCT), Spiral CT, Positron Emission Tomography (PET),Single-Photon Emission Computed Tomography (SPECT), X-ray, opticaltomography, fluorescence imaging, ultrasound imaging, and radiotherapyportal imaging, etc., or the combination thereof. In some embodiments,medical image 102 may be taken with contrast agent to enhance the imagecontrast. In step S404, diagnostic report generating system 100 receivespatient information 104 from patient database 103. For example, patientinformation 104 may include age and gender of the patient, patientmedical history, and family medical history, etc. Although S404 isillustrated as following S402 in FIG. 4 , it is contemplated that S402and S404 can be performed in any order relative to each other, orsimultaneously with each other.

In step S406, diagnostic report generating system 100 detects asuspicious object from medical images 102. In some embodiments, CAD unit122 may be used to perform a CAD analysis to detect the conspicuousobject. The conspicuous object may be a lesion, a tumor, or anotheranatomical structure that is potentially indicative of a medicalcondition of the patient that needs to be treated. For example, theconspicuous object may be a cerebral lesion as shown in FIG. 2 or one ormore lung nodules as shown in FIG. 3 . To detect the conscious object,CAD unit 122 may perform pre-processing, image segmentation, andclassification, etc., as described in detail above in connection withFIG. 1 . In some embodiments, the detected object may be highlighted inmedical image 102 for the radiologist, for example, using boundarycontours or bounding boxes.

In step S408, diagnostic report generating system 100 diagnoses amedical condition of the patient based on the detected object. Themedical condition may be, e.g., that the detected object region isbleeding, or that the detected object is a tumor indicative of a cancer.In some embodiments, diagnostic report generating system 100 maydiagnose the medical condition automatically, e.g., using classificationmethods. For example, the detected object may be classified betweennegative (e.g., tumor) or positive (e.g., benign). Variousclassification algorithms may be used including the Nearest-NeighborRule (e.g. k-nearest neighbors), Minimum distance classifier. Cascadeclassifier, Naive Bayesian Classifier, Artificial Neural Network, Radialbasis function network (RBF), Support Vector Machine (SVM) and PrincipleComponent Analysis (PCA), etc In some embodiments, automatic diagnosismay be performed using machine learning methods, such as based on aneural network trained using sample images and known conditions.Applying the neural network to the detected object, CAD unit 122 maydetermine a medical condition and the probability of the medicalcondition exists within the patient. For example, CAD unit 122 maydetermine the patient has a 90% probability of having lung cancer. Insome embodiments, the diagnosis may be made semi-automatically, e.g.,combined with inputs from radiologists.

In step S410, diagnostic report generating system 100 calculatesquantitative parameters associated with the medical condition. Theparameters quantify and describe the medical condition. In someembodiments, the parameters may include size (such as diameter, length,width, depth, etc.), volume, pixel intensities, or the contrastcharacteristics of a tumor. For example, the diameter of a cerebrallesion in FIG. 3 is 2.5 cm, which indicates severity of the hemorrhagecondition in the left frontal lobe. In some embodiments, the parametersmay be determined partially based on a user input, e.g., from aradiologist. For example, the radiologist may manually measure theparameters using a ruler provided by user interface 200/300.

In step S412, diagnostic report generating system 100 automaticallyconstructs a diagnosis report. In some embodiments, construction of thediagnosis report may be performed by report generation unit 124. In someembodiments, report generation unit 124 may generate automated reportcontent using patient information 104. For example, the report mayinclude a patient information section 212/312 showing patient name,gender, and age, as well as examination section 214 containing scaninformation derived from the patient's meta data. Report generation unit124 may further generate diagnosis content of the report based on stepS408. For example, the diagnosis report may include impression section216/316 and findings section 218/318. The diagnosis sections in thereport may include screenshots of images imported from the CAD analysisas well as text information indicating, e.g., the type of the detectedobject (i.e. bleeding type cerebral hemorrhage), the position of thedetected object (i.e. left frontal lobe), and parameters calculated instep S410. In some embodiments, the text information may be determinedbased on lookup tables such as shown in Tables 1 and 2. In someembodiments, report generation unit 124 may utilize the meta informationto improve the diagnosis results provided by CAD unit 122. For example,CAD unit 122 may detect lung nodules and predict nodule malignancysolely based on medical image 102. As part of step S412, reportgeneration unit 124 may use meta data such as age, smoking history,occupations, etc. to improve the malignancy prediction of the nodule.

In step S414, diagnostic report generating system 100 displays theanalyzed medical images and the automatically constructed diagnosisreport, e.g., on display 130. In some embodiments, the medical imagesand the diagnosis report may be displayed side-by-side. In someembodiments, the medical images are displayed with the detected objectshighlighted, e.g., using boundary contours or bounding boxes or enhancedcolors. In some embodiments, when more than N conspicuous objects aredetected, the top N objects may be displayed.

In step S416, diagnostic report generating system 100 receives userinteraction 106. In some embodiments, user 105 may provide a userinteraction 106 via I/O interface 140. In some embodiments, user 105 mayedit the CAD results. For example, user interaction 106 may be user 105selecting an image slice from a 3D image or selecting a detected objectfrom an object list (e.g., selecting a lung nodule from a nodule list).As another example, user interaction 106 may further include user 105manually drawing a ruler/curve/region/point/bounding box/text in theview by using image editing tools. As yet another example, userinteraction 106 may also include sending or dragging the images shown ina CAD view and/or parameters calculated based on the CAD analysis to beincluded in the diagnosis report. In some embodiments, user interaction106 may also include interactions with the report itself. For example,the diagnosis report may include one or more editable areas. An editablearea in the diagnosis report allows user 105 (e.g., aradiologist/clinician) to add, modify, or delete the report content.

In step S418, diagnostic report generating system 100 automaticallyupdates the diagnosis report based on the user action. In someembodiments, when user 105 selects a different image slice (e.g., from a3D image) or a different detected object (e.g., another lung nodule inthe image), report generation unit 124 accordingly updates thescreenshots included in the diagnosis report. In some embodiments,report generation unit 124 automatically recalculates the quantitativeparameters based on the edits made to the images. User edits to thereport itself may be incorporated or otherwise reconciled with theexisting report.

Another aspect of the disclosure is directed to a non-transitorycomputer-readable medium storing instructions which, when executed,cause one or more processors to perform the methods, as discussed above.The computer-readable medium may include volatile or non-volatile,magnetic, semiconductor, tape, optical, removable, non-removable, orother types of computer-readable medium or computer-readable storagedevices. For example, the computer-readable medium may be the storagedevice or the memory module having the computer instructions storedthereon, as disclosed. In some embodiments, the computer-readable mediummay be a disc or a flash drive having the computer instructions storedthereon.

It will be apparent to those skilled in the art that variousmodifications and variations can be made to the disclosed system andrelated methods. Other embodiments will be apparent to those skilled inthe art from consideration of the specification and practice of thedisclosed system and related methods.

It is intended that the specification and examples be considered asexemplary only, with a true scope being indicated by the followingclaims and their equivalents.

What is claimed is:
 1. A system for generating a diagnosis report based on a medical image of a patient, comprising: a communication interface configured to receive the medical image, wherein the medical image is a computed tomography (CT), magnetic resonance imaging (MRI), functional MRI, cone beam computed tomography (CBCT), spiral CT, positron emission tomography (PET), single-photon emission computed tomography (SPECT), X-ray, optical tomography, fluorescence imaging, ultrasound imaging, or radiotherapy portal imaging image; at least one processor configured to: automatically generate text information describing a medical condition of the patient detected from the medical image; construct the diagnosis report, wherein the diagnosis report includes at least one image view showing the medical condition and a report view including the text information describing the medical condition; and provide the diagnosis report for display.
 2. The system of claim 1, wherein the medical condition is automatically detected by the at least one processor, or identified by a user with assistance of the at least one processor.
 3. The system of claim 1, wherein to automatically generate the text information describing the medical condition, the at least one processor is configured to use a text dictionary storing key words describing image observations.
 4. The system of claim 3, wherein the text dictionary is saved in a lookup table, wherein the at least one processor is configured to query the lookup table for corresponding key words based on the medical condition and generate the text information using the key words.
 5. The system of claim 3, wherein the text dictionary is trained using existing clinical reports, wherein the existing clinical reports are mined to identify sample key words to describe the image observations.
 6. The system of claim 1, wherein to automatically generate the text information describing the medical condition, the at least one processor is configured to use a machine learning network trained to predict at least one sentence describing image observations.
 7. The system of claim 1, wherein the at least one processor is further configured to detect an object associated with the medical condition from the medical image, wherein the text information includes a type of the detected object or a position of the detected object.
 8. The system of claim 1, wherein the at least one processor is further configured to determine one or more quantitative parameters of the medical condition, wherein the text information includes the one or more quantitative parameters of the medical condition.
 9. The system of claim 1, wherein the communication interface is further configured to receive meta data along with the medical image of the patient, wherein the at least one processor is configured to automatically generate patient information based on the meta data, wherein the report view further includes the patient information.
 10. The system of claim 1, wherein the diagnosis report displays the at least one image view and the generated text information side-by-side.
 11. A method for generating a diagnosis report based on a medical image of a patient, comprising: receiving, by a communication interface, the medical image, wherein the medical image is a computed tomography (CT), magnetic resonance imaging (MRI), functional MRI, cone beam computed tomography (CBCT), spiral CT, positron emission tomography (PET), single-photon emission computed tomography (SPECT), X-ray, optical tomography, fluorescence imaging, ultrasound imaging, or radiotherapy portal imaging image; automatically generating, by at least one processor, text information describing a medical condition of the patient detected from the medical image; and constructing, by the at least one processor, the diagnosis report, wherein the diagnosis report includes at least one image view showing the medical condition and a report view including the text information describing the medical condition; and providing the diagnosis report for display.
 12. The method of claim 11, wherein the medical condition is automatically detected by the at least one processor, or identified by a user with assistance of the at least one processor.
 13. The method of claim 11, wherein automatically generating the text information describing the medical condition further comprises: querying a text dictionary for corresponding key words based on the medical condition; and generating the text information using the key words.
 14. The method of claim 13, wherein the text dictionary is trained using existing clinical reports, wherein the existing clinical reports are mined to identify sample key words to describe image observations.
 15. The method of claim 11, wherein automatically generating the text information describing the medical condition includes using a machine learning network trained to predict at least one sentence describing image observations.
 16. The method of claim 11, further comprising detecting an object associated with the medical condition from the medical image, wherein the text information includes a type of the detected object or a position of the detected object.
 17. The method of claim 11, further comprising determining one or more quantitative parameters of the medical condition, wherein the text information includes the one or more quantitative parameters of the medical condition.
 18. The method of claim 11, further comprising: receiving meta data along with the medical image of the patient; and automatically generating patient information based on the meta data, wherein the report view further includes the patient information.
 19. A non-transitory computer-readable medium having a computer program stored thereon, wherein the computer program, when executed by at least one processor, performs a method for generating a diagnosis report based on a medical image of a patient, comprising: receiving the medical image, wherein the medical image is a computed tomography (CT), magnetic resonance imaging (MRI), functional MRI, cone beam computed tomography (CBCT), spiral CT, positron emission tomography (PET), single-photon emission computed tomography (SPECT), X-ray, optical tomography, fluorescence imaging, ultrasound imaging, or radiotherapy portal imaging image; automatically generating text information describing a medical condition of the patient detected from the medical image; and constructing the diagnosis report, wherein the diagnosis report includes at least one image view showing the medical condition and a report view including the text information describing the medical condition; and providing the diagnosis report for display.
 20. The non-transitory computer-readable medium of claim 19, wherein automatically generating the text information describing the medical condition further comprises: querying a text dictionary for corresponding key words based on the medical condition; and generating the text information using the key words. 